Perimeter sensor housings

ABSTRACT

The technology relates to an exterior sensor system for a vehicle configured to operate in an autonomous driving mode. The technology includes a close-in sensing (CIS) camera system to address blind spots around the vehicle. The CIS system is used to detect objects within a few meters of the vehicle. Based on object classification, the system is able to make real-time driving decisions. Classification is enhanced by employing cameras in conjunction with lidar sensors. The specific arrangement of multiple sensors in a single sensor housing is also important to object detection and classification. Thus, the positioning of the sensors and support components are selected to avoid occlusion and to otherwise prevent interference between the various sensor housing elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.16/737,359, filed Jan. 8, 2020, which claims the benefit of the filingdate of U.S. Provisional Application No. 62/954,938, filed Dec. 30,2019, the entire disclosures of which are incorporated by referenceherein. The present application is related to co-pending U.S.application Ser. No. 16/737,263, entitled Close-in Sensing CameraSystem, filed Jan. 8, 2020, which claims the benefit of the filing dateof U.S. Provisional Application No. 62/954,930, filed Dec. 30, 2019, theentire disclosures of which are incorporated by reference herein.

BACKGROUND

Self-driving (autonomous) vehicles do not require a human driver in someor all situations. Such vehicles can transport passengers or cargo fromone location to another. They may operate in a fully autonomous mode ora partially autonomous mode where a person may provide some drivinginput. In order to operate in an autonomous mode, the vehicle may employsensors for detecting vehicles and other objects in its externalenvironment, and use received information from the sensors to performvarious driving operations. However, objects immediately adjacent to thevehicle and occlusions in the sensors' fields of view may adverselyimpact driving operations.

BRIEF SUMMARY

The technology relates to an exterior sensor system for a vehicleconfigured to operate in a self-driving (autonomous) mode. Generally,sensors are used to detect objects in the environment around thevehicle. These can include lidar, radar, cameras, sonar and/or othersensors. Different sensors have various benefits, and sensor fusion frommultiple sensors can be employed to obtain a more complete understandingof the environment so that the vehicle can make driving decisions.However, depending on the size, shape, etc. of the vehicle and objectsin the environment, blind spots can exist that can impact drivingdecisions and other autonomous operations. These include blind spotsimmediately adjacent to the vehicle. Such issues can be substantiallymitigated by careful selection and positioning of sensor housings thatmay co-locate different types of sensors in an integrated unit. This caninclude a close-in camera system integrated with a lidar sensor,perimeter view cameras collocated with radar and/or other sensors, etc.

According to one aspect, an external sensing module for a vehicleconfigured to operate in an autonomous driving mode is provided. Theexternal sensing module comprises a lidar sensor and an image sensor.The lidar sensor is arranged along a first housing section of theexternal sensing module. The lidar sensor is configured to detectobjects in a region of an external environment around the vehicle andwithin a threshold distance of the vehicle. The image sensor is arrangedalong a second housing section of the external sensing module. Thesecond housing section with the image sensor is vertically aligned withthe first housing section with the lidar sensor. The image sensor isarranged along the vehicle to have an unobstructed field of view of theregion of the external environment. The image sensor provides a selectedresolution for objects within the threshold distance of the vehicle toclassify objects detected by the lidar sensor. A third housing sectionof the external sensing module is disposed between the first and secondhousing sections. The third housing section includes an exterior surfacearranged to prevent interference between the image sensor and the lidarsensor.

The image sensor can be arranged above or below the lidar sensor. Theimage sensor may be arranged at a downward angle relative to a side ofthe vehicle, in which the downward angle is on the order of 20-40° toprovide coverage within the threshold distance of the vehicle. Thethreshold distance may be between 1-3 meters from the vehicle.

The external sensing module may further comprise one or more illuminatorunits configured to illuminate the field of view of the image sensor. Inone example, the one or more illuminator units may be disposed adjacentto the image sensor along the second housing section. For instance, theone or more illuminator units can be a pair of illuminators disposed oneither side of the image sensor. The external sensing module may furtherinclude a cleaning mechanism configured to clean the image sensor and/orthe one or more illuminator units.

In an example, the external sensing module further comprises a radarsensor disposed along a fourth housing section of the external sensingmodule arranged to one side of both the first and second housingsections. The fourth housing section may be arranged closer to a frontof the vehicle than the first and second sections. The external sensingmodule may further comprise at least one additional image sensordisposed along a fifth housing section of the external sensing module,in which the fourth housing section is arranged between the fifthhousing section and the first and second housing sections. In oneexample, the at least one additional image sensor comprises a givenadditional image sensor disposed generally planar to the radar sensorand arranged to provide a selected field of view along a side of thevehicle separate from the field of view of the image sensor arrangedalong the second housing section. In another example, the at least oneadditional image sensor comprises a given additional image sensordisposed generally perpendicular to the radar sensor and arranged toprovide a selected field of view along a front of the vehicle separatefrom the field of view of the image sensor arranged along the secondhousing section. The at least one additional image sensor may includefirst and second additional image sensors. Here, the first additionalimage sensor is disposed generally planar to the radar sensor andarranged to provide a selected field of view along a side of the vehicleseparate from the field of view of the image sensor arranged along thesecond housing section, And the second additional image sensor isdisposed generally perpendicular to the radar sensor and the firstadditional image sensor and arranged to provide a selected field of viewalong a front of the vehicle separate from the field of view of theimage sensor arranged along the second housing section and the selectedfield of view of the first additional image sensor.

According to another aspect, an external sensing module for a vehicleconfigured to operate in an autonomous driving mode is provided. Theexternal sensing module comprises a radar sensor and an image sensor.The radar sensor is arranged along a first housing section of theexternal sensing module. The radar sensor is configured to detectobjects in a region of an external environment around the vehicle andwithin a threshold distance of the vehicle. The image sensor is arrangedalong a second housing section of the external sensing module. Thesecond housing section with the image sensor is adjacent to the firsthousing section with the radar sensor. The image sensor is arrangedalong the vehicle to have an overlapping field of view of the region ofthe external environment. The image sensor provides a selectedresolution for objects within the threshold distance of the vehicle toclassify objects detected by the radar sensor.

The external sensing module may be disposed along a rear fascia of thevehicle to detect and classify objects behind the vehicle. For instance,the field of view of the image sensor may be between 30-60° along therear of the vehicle. Alternatively or additionally, the field of view ofthe image sensor has an outer azimuth of between 15-35° and an innerazimuth of between 10-25°. The second housing section may be disposedcloser to an adjacent side of the vehicle than the first housingsection.

According to yet another aspect, a vehicle configured to operate in anautonomous driving mode is provided. The vehicle comprises a drivingsystem and an external sensing module. The driving system includes adeceleration system configured to control braking of the vehicle, anacceleration system configured to control acceleration of the vehicle,and a steering system configured to control wheel orientation and adirection of the vehicle. The external sensing module includes a lidarsensor and an image sensor. The lidar sensor is arranged along a firsthousing section of the external sensing module. The lidar sensor isconfigured to detect objects in a region of an external environmentaround the vehicle and within a threshold distance of the vehicle. Theimage sensor is arranged along a second housing section of the externalsensing module. The second housing section with the image sensor isvertically aligned with the first housing section with the lidar sensor.The image sensor is arranged along the vehicle to have an unobstructedfield of view of the region of the external environment. The imagesensor provides a selected resolution for objects within the thresholddistance of the vehicle to classify objects detected by the lidarsensor. A third housing section of the external sensing module isdisposed between the first and second housing sections. The thirdhousing section includes an exterior surface arranged to preventinterference between the image sensor and the lidar sensor.

The external sensing module may further comprise one or more illuminatorunits configured to illuminate the field of view of the image sensor.The one or more illuminator units are disposed adjacent to the imagesensor along the second housing section. The external sensing module mayfurther include a cleaning mechanism configured to clean the imagesensor and/or the one or more illuminator units.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-B illustrate an example vehicle configured for use with aspectsof the technology.

FIG. 1C illustrates another example vehicle configured for use withaspects of the technology.

FIGS. 1D-E illustrate an example cargo vehicle configured for use withaspects of the technology.

FIG. 2A is a block diagram of systems of an example vehicle inaccordance with aspects of the technology.

FIGS. 2B-C are block diagrams of systems of an example cargo-typevehicle in accordance with aspects of the technology.

FIG. 3 illustrates examples of regions around a vehicle in accordancewith aspects of the disclosure.

FIG. 4 illustrates example sensor fields of view in accordance withaspects of the disclosure.

FIG. 5 illustrates example perimeter camera fields of view in accordancewith aspects of the disclosure.

FIGS. 6A-C illustrate example arrangements of perimeter cameras andinfrared illuminators in accordance with aspects of the disclosure.

FIGS. 7A-F illustrate an example perimeter sensor housing assembly inaccordance with aspects of the disclosure.

FIG. 8 illustrates an example sensor arrangement to minimize occlusionsin accordance with aspects of the disclosure.

FIG. 9 illustrates an example of perimeter sensor fields of view inaccordance with aspects of the disclosure.

FIG. 10 illustrates an example occlusion scenario in accordance withaspects of the technology.

FIGS. 11A-C illustrate an example perimeter sensor housing assembly inaccordance with aspects of the disclosure.

FIG. 12 illustrates an example sensor arrangement to minimize occlusionsin accordance with aspects of the disclosure.

FIG. 13 illustrates an example of perimeter sensor fields of view inaccordance with aspects of the disclosure.

FIGS. 14A-E illustrate another example perimeter sensor housing assemblyin accordance with aspects of the disclosure.

FIG. 15 illustrates a variation of the perimeter sensor housing assemblyof FIGS. 14A-E.

FIGS. 16A-E illustrate yet another example perimeter sensor housingassembly in accordance with aspects of the disclosure.

FIG. 17 illustrates a variation of the perimeter sensor housing assemblyof FIGS. 16A-E.

FIG. 18 illustrates a method of operation in accordance with aspects ofthe disclosure.

DETAILED DESCRIPTION

Aspects of the technology involve a close-in sensing (CIS) camera systemto address blind spots around the vehicle. The CIS system is used tohelp classify objects detected within a few meters (e.g., less than 3meters) of the vehicle. Based on object classification, the system isable to distinguish between objects that may be “driveable” (somethingthe vehicle can drive over) versus “non-drivable”. By way of example, adriveable object could be vegetation, a pile of leaves, a paper orplastic bag, etc., while non-driveable objects would include those typesof objects that either must be avoided (e.g., pedestrians, bicyclists,pets, etc.) or that may damage the vehicle if driven over (e.g., tallcurbs, broken glass, deep potholes, fire hydrant, etc.) In one scenario,classification is enhanced by employing cameras in conjunction withlidar sensors. This can be very important when trying to determinewhether a person is next to the vehicle. The cameras may each includeone or more image sensors. The image sensors may be CMOS sensors,although CCD or other types of imaging elements may be employed.

Other aspects of the technology relate to the arrangements andconfigurations of multiple sensors in a single sensor housing. Asdiscussed further below, there are advantages to co-locating differentsensor types in the same housing, for instance to aid in sensor fusion.However, the positioning of the sensors can be very important, forinstance to avoid occlusion of one sensor by another, to ensure that thecalibration between the sensors is more accurate, and/or to otherwiseprevent interference between the sensors. By way of example, anilluminator, such as an infrared (IR) or optical illuminator, should bearranged to avoid shining its light directly into the lens of a camera,for instance a camera that is sensitive to IR light.

Example Vehicle Systems

FIG. 1A illustrates a perspective view of a passenger vehicle 100, suchas a minivan, sedan or sport utility vehicle. FIG. 1B illustrates atop-down view of the passenger vehicle 100. The passenger vehicle 100may include various sensors for obtaining information about thevehicle's external environment. For instance, a roof top housing 102 mayinclude a lidar sensor as well as various cameras, radar units, infraredand/or acoustical sensors. Housing 104, located at the front end ofvehicle 100, and housings 106 a, 106 b on the driver's and passenger'ssides of the vehicle may each incorporate a lidar and/or other sensors.For example, housing 106 a may be located in front of the driver's sidedoor along a quarterpanel of the vehicle. As shown, the passengervehicle 100 also includes housings 108 a, 108 b for radar units, lidarand/or cameras also located towards the rear roof portion of thevehicle. Other housings 110 a, 110 b may be located along the rearquarterpanels, for instance above and behind the rear wheels.

Additional lidar, radar units and/or cameras (not shown) may be locatedat other places along the vehicle 100. For instance, arrow 112 indicatesthat a sensor unit (112 in FIG. 1B) may be positioned along the rear ofthe vehicle 100, such as on or adjacent to the bumper or trunk door/lid.And arrow 114 indicates a series of sensor units 116 arranged along aforward-facing direction of the vehicle. While shown separately, in oneexample, the sensor units 116 may be integrated into a front-facingsection of the rooftop housing 102. In some examples, the passengervehicle 100 also may include various sensors for obtaining informationabout the vehicle's interior spaces. The interior sensor(s) may includeat least one of a camera sensor, an auditory sensor and an infraredsensor.

Depending on the vehicle type and configuration, more or fewer sensorhousings may be disposed around the vehicle. For instance, as shown withthe example vehicle 150 of FIG. 1C, similar to vehicle 100 there may bea roof top sensor housing 152, front sensor housing 154, side housings156 a and 156 b along front quarterpanels, side housings 158 along rearquarterpanels, and a rear sensor housing indicated by arrow 160. Whilecertain aspects of the disclosure may be particularly useful inconnection with specific types of vehicles, the vehicle may be any typeof vehicle including, but not limited to, cars, trucks, motorcycles,buses, recreational vehicles, etc.

FIGS. 1D-E illustrate an example cargo vehicle 170, such as atractor-trailer truck. The truck may include, e.g., a single, double ortriple trailer, or may be another medium or heavy duty truck such as incommercial weight classes 4 through 8. As shown, the truck includes atractor unit 172 and a single cargo unit or trailer 174. The trailer 174may be fully enclosed, open such as a flat bed, or partially opendepending on the type of cargo to be transported. In this example, thetractor unit 172 includes the engine and steering systems (not shown)and a cab 176 for a driver and any passengers.

The trailer 174 includes a hitching point, known as a kingpin, 178. Thekingpin 178 is typically formed as a solid steel shaft, which isconfigured to pivotally attach to the tractor unit 172. In particular,the kingpin 178 attaches to a trailer coupling 180, known as afifth-wheel, that is mounted rearward of the cab. For a double or tripletractor-trailer, the second and/or third trailers may have simple hitchconnections to the leading trailer. Or, alternatively, each trailer mayhave its own kingpin. In this case, at least the first and secondtrailers could include a fifth-wheel type structure arranged to coupleto the next trailer.

As shown, the tractor may have one or more sensor units 182, 184 and/or186 disposed therealong. For instance, one or more sensor units 182 maybe disposed on a roof or top portion of the cab 176, and one or moreside sensor units 184 may be disposed on left and/or right sides of thecab 176. Sensor units may also be located along other regions of the cab176, such as along the front bumper or hood area, in the rear of thecab, adjacent to the fifth-wheel, underneath the chassis, etc. Thetrailer 174 may also have one or more sensor units 186 disposedtherealong, for instance along a side panel, front, rear, roof and/orundercarriage of the trailer 174.

As with the sensor units of the passenger vehicle of FIGS. 1A-B, eachsensor unit of the cargo vehicle may include one or more sensors, suchas lidar, radar, camera (e.g., optical or infrared), acoustical (e.g.,microphone or sonar-type sensor), inertial (e.g., accelerometer,gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPSsensors). While certain aspects of the disclosure may be particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, buses, recreational vehicles, etc.

FIG. 2A illustrates a block diagram 200 with various components andsystems of exemplary vehicles, such as vehicles 100 and 150, configuredto operate in a fully or semi-autonomous mode of operation. By way ofexample, there are different degrees of autonomy that may occur for avehicle operating in a partially or fully autonomous driving mode. TheU.S. National Highway Traffic Safety Administration and the Society ofAutomotive Engineers have identified different levels to indicate howmuch, or how little, the vehicle controls the driving. For instance,Level 0 has no automation and the driver makes all driving-relateddecisions. The lowest semi-autonomous mode, Level 1, includes some driveassistance such as cruise control. Level 2 has partial automation ofcertain driving operations, while Level 3 involves conditionalautomation that can enable a person in the driver's seat to take controlas warranted. In contrast, Level 4 is a high automation level where thevehicle is able to drive without assistance in select conditions. AndLevel 5 is a fully autonomous mode in which the vehicle is able to drivewithout assistance in all situations. The architectures, components,systems and methods described herein can function in any of the semi orfully-autonomous modes, e.g., Levels 1-5, which are referred to hereinas “autonomous” driving modes. Thus, reference to an autonomous drivingmode includes both partial and full autonomy.

As illustrated in FIG. 2 , the block diagram 200 includes one or morecomputing devices 202, such as computing devices containing one or moreprocessors 204, memory 206 and other components typically present ingeneral purpose computing devices. The memory 206 stores informationaccessible by the one or more processors 204, including instructions 208and data 210 that may be executed or otherwise used by the processor(s)204. The computing system may control overall operation of the vehiclewhen operating in an autonomous mode.

The memory 206 stores information accessible by the processors 204,including instructions 208 and data 210 that may be executed orotherwise used by the processor 204. The memory 206 may be of any typecapable of storing information accessible by the processor, including acomputing device-readable medium. The memory is a non-transitory mediumsuch as a hard-drive, memory card, optical disk, solid-state, etc.Systems may include different combinations of the foregoing, wherebydifferent portions of the instructions and data are stored on differenttypes of media.

The instructions 208 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The data 210 may be retrieved, stored ormodified by one or more processors 304 in accordance with theinstructions 208. In one example, some or all of the memory 206 may bean event data recorder or other secure data storage system configured tostore vehicle diagnostics, detected sensor data and/or one or morebehavior/classification models used in conjunction with object detectionand classification, which may be on board the vehicle or remote,depending on the implementation. For instance, the models may be used toclassify whether an object is a person (e.g., a pedestrian), a bicycle,a ball or a construction sign that is adjacent to the vehicle. Based onthe classification, the system may predict or assign a behavior for thatobject, and use the classification/behavior when making adriving-related decision. This can include, for instance, alerting apedestrian next to the vehicle that the vehicle is on and planning toexit a parking spot.

The processors 204 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 2 functionally illustrates the processors, memory, andother elements of computing devices 202 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 206 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 204. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 202 may form an autonomous drivingcomputing system incorporated into the vehicle. The autonomous drivingcomputing system may be capable of communicating with various componentsof the vehicle. For example, the computing devices 202 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 212 (for controlling braking ofthe vehicle), acceleration system 214 (for controlling acceleration ofthe vehicle), steering system 216 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 218 (forcontrolling turn signals), navigation system 220 (for navigating thevehicle to a location or around objects) and a positioning system 222(for determining the position of the vehicle). The autonomous drivingcomputing system may operate in part as a planner, in accordance withthe navigation system 220 and the positioning system 222, e.g., fordetermining a route from a starting point to a destination.

The computing devices 202 are also operatively coupled to a perceptionsystem 224 (for detecting objects in the vehicle's environment), a powersystem 226 (for example, a battery and/or gas or diesel powered engine)and a transmission system 230 in order to control the movement, speed,etc., of the vehicle in accordance with the instructions 208 of memory206 in an autonomous driving mode which does not require or needcontinuous or periodic input from a passenger of the vehicle. Some orall of the wheels/tires 228 are coupled to the transmission system 230,and the computing devices 202 may be able to receive information abouttire pressure, balance and other factors that may impact driving in anautonomous mode. The power system 226 may have one or more powerdistribution elements, each of which may be capable of supplying powerto selected components and other systems of the vehicle.

The computing devices 202 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 202 may navigate the vehicle to a destination locationcompletely autonomously using data from the map information andnavigation system 220. Computing devices 202 may use the positioningsystem 222 to determine the vehicle's location and the perception system224 to detect and respond to objects when needed to reach the locationsafely. In order to do so, computing devices 202 may cause the vehicleto accelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 214), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 212), change direction (e.g., by turning thefront or other wheels of the vehicle by steering system 216), and signalsuch changes (e.g., by lighting turn signals of signaling system 218).Thus, the acceleration system 214 and deceleration system 212 may be apart of a drivetrain or other type of transmission system 230 thatincludes various components between an engine of the vehicle and thewheels of the vehicle. Again, by controlling these systems, computingdevices 202 may also control the transmission system 230 of the vehiclein order to maneuver the vehicle autonomously.

Navigation system 220 may be used by computing devices 202 in order todetermine and follow a route to a location. In this regard, thenavigation system 220 and/or memory 206 may store map information, e.g.,highly detailed maps that computing devices 202 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and/or right lane lines or other lanemarkers that define the boundary of the lane. Thus, most lanes may bebounded by a left edge of one lane line and a right edge of another laneline.

The perception system 224 includes sensor units for detecting objectsexternal to the vehicle. The detected objects may be other vehicles,obstacles in the roadway, traffic signals, signs, trees, pedestrians,bicyclists, etc. As discussed further below, exterior sensor suite 232includes various housings each having one or more sensors to detectobjects and conditions in the environment external to the vehicle. Andinterior sensor suite 234 may employ one or more other sensors to detectobjects and conditions within the vehicle, such as passengers, pets andpackages in the passenger compartment, packages or other cargo in thetrunk area, etc. For both the exterior sensor suite 232 and the interiorsensor suite 234, the housings having different sensors are disposedabout the vehicle to provide not only object detection in variousenvironmental conditions, but also to enable rapid classification ofdetected objects. This allows the vehicle to make effective real timedriving decisions.

The raw data from the sensors and the aforementioned characteristics canbe processed by the perception system 224 and/or sent for furtherprocessing to the computing devices 202 periodically and continuously asthe data is generated by the perception system 224. Computing devices202 may use the positioning system 222 to determine the vehicle'slocation and perception system 224 to detect and respond to objects whenneeded to reach the location safely. In addition, the computing devices202 may perform calibration of individual sensors, all sensors in aparticular sensor assembly (housing), or between sensors in differentsensor assemblies or other physical housings.

In one example, an external sensor housing may be arranged as a sensortower integrated into a side-view mirror on the vehicle. In anotherexample, other sensors may be part of the roof top housing 102 or 152,or other housings as illustrated in FIGS. 1A-C. The computing devices202 may communicate with the sensor assemblies located on or otherwisedistributed along the vehicle. Each assembly may have one or more typesof sensors such as those described above.

Returning to FIG. 2 , computing devices 202 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 236. The user interface subsystem 236 may include one or moreuser inputs 238 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 240 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (not shown) and may be used bycomputing devices 202 to provide information to passengers within thevehicle. Other output devices such as speaker(s) 242, and input devices244 such as touch screen or buttons may also be located within thepassenger vehicle.

The vehicle also includes a communication system 246. For instance, thecommunication system 246 may also include one or more wireless networkconnections to facilitate communication with other computing devices,such as passenger computing devices within the vehicle, and computingdevices external to the vehicle such as in another nearby vehicle on theroadway or a remote server system. The network connections may includeshort range communication protocols such as Bluetooth™, Bluetooth™ lowenergy (LE), cellular connections, as well as various configurations andprotocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP, and various combinations of the foregoing.

FIG. 2B illustrates a block diagram 250 with various components andsystems of a vehicle, e.g., vehicle 170 of FIGS. 1D-E. By way ofexample, the vehicle may be a truck, farm equipment or constructionequipment, configured to operate in one or more partially autonomousmodes of operation. As shown in the block diagram 250, the vehicleincludes a control system of one or more computing devices similar tothat described above, such as computing devices 202′ containing one ormore processors 204′ and memory 206′ storing instructions 208′ and data210′ such as vehicle diagnostics, detected sensor data and/or one ormore behavior/classification models used in conjunction with objectdetection and classification. In this example, the control system mayconstitute an electronic control unit (ECU) of a tractor unit of a cargovehicle.

In one example, the computing devices may form a driving computingsystem incorporated into vehicle 170. Similar to the arrangementdiscussed above regarding FIG. 2A, the driving computing system of blockdiagram 250 may be capable of communicating with various components ofthe vehicle in order to perform driving operations. For example, thecomputing devices 202′ may be in communication with various systems ofthe vehicle, such as the driving system including a deceleration system212′, acceleration system 214′, steering system 216′, signaling system218′, navigation system 220′ and a positioning system 222′, each ofwhich may function as discussed above regarding FIG. 2A.

The computing devices 302 are also operatively coupled to a perceptionsystem 224′, a power system 226′ and a transmission system 230′. Some orall of the wheels/tires 228′ are coupled to the transmission system230′, and the computing devices 202′ may be able to receive informationabout tire pressure, balance, rotation rate and other factors that mayimpact driving. As with computing devices 202, the computing devices202′ may control the direction and speed of the vehicle by controllingvarious components. By way of example, computing devices 202′ may aidnavigating the vehicle to a destination location using data from the mapinformation and navigation system 220′.

Similar to perception system 224, the perception system 224′ alsoincludes one or more sensors or other components such as those describedabove for detecting objects external to the vehicle, objects orconditions internal to the vehicle, and/or operation of certain vehicleequipment such as the wheels and deceleration system 212′. For instance,as indicated in FIG. 2B the perception system 224′ includes one or moresensor assemblies 252. Each sensor assembly 252 includes one or moresensors. In one example, the sensor assemblies 252 may be arranged assensor towers integrated into the side-view mirrors on the truck, farmequipment, construction equipment or the like. Sensor assemblies 252 mayalso be positioned at different locations on the tractor unit 172 or onthe trailer 174, as noted above with regard to FIGS. 1D-E. The computingdevices 202′ may communicate with the sensor assemblies located on boththe tractor unit 172 and the trailer 174. Each assembly may have one ormore types of sensors such as those described above.

Also shown in FIG. 2B is a coupling system 254 for connectivity betweenthe tractor unit and the trailer. The coupling system 254 may includeone or more power and/or pneumatic connections (not shown), and afifth-wheel 256 at the tractor unit for connection to the kingpin at thetrailer. A communication system 246′, equivalent to communication system246, is also shown as part of vehicle system 250. Similarly, userinterface 236′, equivalent to user interface 236 may also be includedfor interactions with the driver and any passengers of the vehicle.

FIG. 2C illustrates an example block diagram 260 of systems of thetrailer, such as trailer 174 of FIGS. 1D-E. As shown, the systemincludes an ECU 262 of one or more computing devices, such as computingdevices containing one or more processors 264, memory 266 and othercomponents typically present in general purpose computing devices. Thememory 266 stores information accessible by the one or more processors264, including instructions 268 and data 270 that may be executed orotherwise used by the processor(s) 264. The descriptions of theprocessors, memory, instructions and data from FIGS. 2A-B apply to theseelements of FIG. 2C.

The ECU 262 is configured to receive information and control signalsfrom the trailer unit. The on-board processors 264 of the ECU 262 maycommunicate with various systems of the trailer, including adeceleration system 272, signaling system 274, and a positioning system276. The ECU 262 may also be operatively coupled to a perception system278 with one or more sensors for detecting objects in the trailer'senvironment and a power system 280 (for example, a battery power supply)to provide power to local components. Some or all of the wheels/tires282 of the trailer may be coupled to the deceleration system 272, andthe processors 264 may be able to receive information about tirepressure, balance, wheel speed and other factors that may impact drivingin an autonomous mode, and to relay that information to the processingsystem of the tractor unit. The deceleration system 272, signalingsystem 274, positioning system 276, perception system 278, power system280 and wheels/tires 282, as well as sensor assemblies 284, may operatein a manner such as described above with regard to FIGS. 2A-B.

The trailer also includes a set of landing gear 286, as well as acoupling system 288. The landing gear provide a support structure forthe trailer when decoupled from the tractor unit. The coupling system288, which may be a part of coupling system 254, provides connectivitybetween the trailer and the tractor unit. Thus, the coupling system 288may include a connection section 290 (e.g., for power and/or pneumaticlinks). The coupling system also includes a kingpin 292 configured forconnectivity with the fifth-wheel of the tractor unit.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various implementations will now bedescribed in accordance with aspects of the technology.

The environment around the vehicle can be viewed as having differentquadrants or regions. One example 300 is illustrated in FIG. 3 , whichshows front, rear, right side and left side regions, as well as adjacentareas for the front right, front left, right rear and left rear areasaround the vehicle. These regions are merely exemplary. The vehicle'sperception system may cover some or all of the regions around thevehicle to provide as much information as possible about objects in thevehicle's external environment.

For instance, various sensors may be located at different places aroundthe vehicle (see FIGS. 1A-C) to gather data from some or all of theseregions. By way of example, the three sensors 116 of FIG. 1 mayprimarily receive data from the front, front left and front rightregions around the vehicle. In contrast, the roof top housing 102 mayinclude other sensors, such as multiple cameras and/or rotating lidar orradar sensors, to provide a 360° field of view (FOV) around the vehicle.

Certain sensors may have different fields of view depending on theirplacement around the vehicle and the type of information they aredesigned to gather. For instance, different lidar sensors may be usedfor near (short range) detection of objects adjacent to the vehicle(e.g., less than 2-10 meters), while others may be used for far (longrange) detection of objects a hundred meters (or more or less) in frontof the vehicle. Mid-range lidars may also be employed, for instance todetect objects between 10-100 meters from the vehicle. Multiple radarunits may be positioned toward the front, rear and/or sides of thevehicle for short or long-range object detection. Cameras may bearranged to provide good visibility around the vehicle. Depending on theconfiguration, certain sensor housings may include multiple individualsensors with overlapping fields of view. Alternatively, other sensorsmay provide redundant 360° fields of view.

FIG. 4 provides one example 400 of sensor fields of view relating to thesensors illustrated in FIGS. 1A-B. Here, should the roof-top housing 102include a lidar sensor as well as various cameras, radar units, infraredand/or acoustical sensors, each of those sensors may have a differentfield of view. Thus, as shown, the lidar sensor may provide a 360° FOV402, while cameras arranged within the housing 102 may have individualFOVs 404, for instance covering one or more regions about the vehicle asshown in FIG. 3 . A sensor within housing 104 at the front end of thevehicle has a forward facing FOV 406. The housings 106 a, 106 b on thedriver's and passenger's sides of the vehicle may each incorporatelidar, radar, camera and/or other sensors. For instance, lidars withinhousings 106 a and 106 b may have respective FOVs 406 a and 406 b, whileradar units, cameras and/or other sensors within housings 106 a and 106b may have a respective FOV 407 a and 407 b. Similarly, sensors withinhousings 108 a, 108 b located towards the rear roof portion of thevehicle each have a respective FOV. For instance, lidars within housings108 a and 108 b may have a respective FOV 408 and 408 b, while radarunits, cameras and/or other sensors within housings 108 a and 108 b mayhave a respective FOV 409 a and 409 b. The sensors in housings 110 a and110 b towards the rear of the vehicle may have respective fields of view410 a and 410 b. The sensors within housing 112 at the rear end may havea rearward facing FOV 412. And the series of sensor units 116 arrangedalong a forward-facing direction of the vehicle may have respective FOVs414, 416 and 418. Each of these fields of view is merely exemplary andnot to scale in terms of coverage range. And while only one or two FOVsare shown associated with a given sensor housing, depending on thenumber of sensors and their configuration, more (or fewer) fields ofview may be associated with that sensor housing.

As discussed further below, collocating different types of sensors inthe same housing can provide enhanced object detection and enable theonboard system to rapidly classify detected objects. The collocatedsensors may be the same or substantially overlapping fields of view, orotherwise provide complementary fields of view.

Example Scenarios

The elevation and orientation of the camera, lidar, radar and/or othersensor subsystems will depend on placement of the various housings onthe vehicle, as well as the type of vehicle. For instance, if a sensorhousing is mounted on or above the roof of a large SUV (e.g., vehicle100), the elevation will typically be higher than when the housing ismounted on the roof of a sedan or sports car (e.g., vehicle 150). Also,the visibility may not be equal around all areas of the vehicle due toplacement and structural limitations. By varying the placement on thevehicle, a suitable field of view can be obtained for the sensors ineach housing. This can be very important for detecting objectsimmediately adjacent to the vehicle (e.g., within 1-2 meters or no morethan 3 meters from the vehicle), as well as for objects farther from thevehicle. There may be requirements for detecting adjacent and remoteobjects in various scenarios, such as checking the immediate vicinitybefore pulling out of a parking space, determining whether to make anunprotected left turn, etc.

Close Sensing Camera System

In view of the above, aspects of the technology provide a close sensingcamera system as part of the sensor suite for objects within a thresholddistance of the vehicle. This camera system is designed to prevent thevehicle from being stuck (when not moving) or acting awkwardly in caseswhere the self-driving system cannot distinguish between a driveable andnon-driveable object within a certain distance of the vehicle. By way ofexample, the close sensing camera system is configured to provide sensorinformation for objects within a threshold distance of, e.g., no morethan 6-10 meters from the vehicle. In some instances, the thresholddistance may be no more than 2-3 meters from the vehicle. Thisinformation is used to help detect and classify objects, such aspedestrians standing next to the vehicle, bicycles or motorcycles parkedadjacent to the vehicle, and balls, construction signs or other objectsthat may be in the nearby vicinity.

Lidar sensors may be arranged around the vehicle to minimize blind spotsand detect objects. Such sensors are very capable of detecting thepresence of objects. However, sensor data from a lidar (e.g., a lidarpoint cloud) by itself may not be sufficient for the self-driving systemto determine what kind of object is present. When it is unclear whattype of object is nearby, the vehicle could employ a conservativebehavior, such as waiting a few minutes to observe around the vehicle,honk its horn, blink its lights, etc., to see how the object reacts, orbacking up or edging forward slowly to obtain a clearer picture of thesurroundings. However, this may not provide additional usefulinformation about the object and could irritate or cause confusion forpassengers, nearby pedestrians and other road users.

Thus, according to one aspect of the technology, one or more cameras canbe arranged with the lidar sensor in a single sensor housing to enablerapid classification of an object, for instance to determine if it is apedestrian, bicycle or traffic cone. The camera field of view mayencompass, and in certain examples be larger than, the lidar field ofview. This can be accomplished with one camera or multiple camerashaving complementary or otherwise overlapping fields of view. By way ofexample, a person may be standing or sitting next to the vehicle. Thismay occur, for instance, when the person exits the vehicle, appears frombehind a nearby parked car, or is already in a blind spot before thevehicle turns on or prepares to exit a parking space. Other scenarioswhere this camera system is beneficial include unprotected turns, highspeed lane changes, occlusions of oncoming traffic by other objects, lowmounted metering lights (such as at an on-ramp of a freeway),identifying road cones and other construction items, and detecting smallforeign object debris (FOD).

Classification of a detected object may include determining the size,proximity and orientation of the detected object. The system isconfigured so that the cameras are able to see a minimum thresholdvolume taken up by the object of interest (e.g., at least 50% of acuboid or other 3D shape). In one example, each camera of the closesensing system is co-located with a companion lidar sensor. Forinstance, the camera may be no more than 1 foot or 0.3 meters from thelidar sensor, such as to avoid parallax. The camera may be mounted tothe vehicle using the same bracket or housing as the lidar, or they maybe mounted separately. In general operation, the system FOV shouldprovide a 3600 view around the vehicle up to 3 meters away.

The camera resolution should be sufficient to satisfy a thresholdclassification based on a minimum number of pixels. By way of example,one classification threshold may be the ability to classify a particularobject of a selected cuboid shape using no more than 32-64 pixels whenthe object is within 3 meters of the vehicle. Or, alternatively, thethreshold classification may necessitate a camera having a resolutionrequirement of between 0.1-0.4 mrad/pixel. The threshold classificationrequirements may vary, for instance depending on the type of object andscenario (e.g., is there an adult or child standing or sitting next tothe vehicle, is there a motorcyclist approaching from behind the vehicleover 100 m away, etc.).

The cameras may provide different, and potentially overlapping, zones ofcoverage, as shown in example 500 of FIG. 5 . In this example, up to 8(or more) cameras may be employed to provide front, side and rear facingFOVs. By way of example, FOVs 502 a and 502 b encompass portions of thefront left and front right regions around the vehicle. FOVs 504 a and504 b overlap with FOVs 502 a and 502 b, providing additional coveragealong the front, front left and front right regions. FOV 506 providescovers in the front region of the vehicle. FOVs 508 a and 508 b providecoverage facing towards the rear of the vehicle, for instance along theleft/right regions and rear left and rear right regions. And FOV 510provides coverage along the rear region of the vehicle.

The camera may be required to operate in all ambient lightingsituations. As such, different cameras may rely on illumination fromvehicle sources (e.g., the headlights, parking lights, backup lights,running lights) and environmental sources (e.g., other vehicles,streetlights, etc.) Alternatively or additionally, an IR and/or opticallight illuminator may be arranged to provide illumination to the camera.For instance, one or more illuminators may be arranged adjacent to thecamera on the sensor housing.

By way of example, cameras with front and rear facing FOVs may notrequire separate illumination, as headlights and brake lights or backuplights may provide sufficient lighting in some scenarios. However,cameras with side FOVs may require supplemental illumination inlow-light conditions. Such supplemental lighting may be provided via anear-infrared (NIR) emitter placed near the camera. As discussed furtherbelow, in one configuration a pair of “saddle” illuminator modules (forinstance NIR modules) may be employed on either side of a camera, whereeach illuminator module compensates for occlusions of the other module.Alternatively, a single monolithic illuminator module may be used.

FIGS. 6A-6C illustrate exemplary camera and illuminator configurations.In particular, FIG. 6A illustrates a first configuration 600, in whichthe illuminators are disposed to the side of the respective cameras.Here, four cameras are shown around the vehicle: front facing camera602, rear facing camera 604 and left and right side facing cameras 606 aand 606 b, respectively. As shown, a pair of illuminators 608 a and 608b are arranged on either side of front camera 602. Similarly, a pair ofsaddle illuminators 610 a and 610 b are arranged on either side of rearcamera 604. However, in an alternative configuration, only oneilluminator may be arranged to the side of front camera 602 and/or rearcamera 604. Side cameras 606 a and 606 b are shown each having arespective illuminator 612 a, 612 b disposed to the side thereof. Inthese examples, the illuminators 612 are positioned rearward of the sidecameras 606. Here, the side cameras 606 may receive some illuminationfrom the front illuminators 608 a and 608 b, which can supplement IRillumination from the illuminators 612 a, 612 b.

FIG. 6B illustrates a second configuration 650, in which a singleilluminator is disposed above the respective camera. As with FIG. 6A,four cameras are shown around the vehicle: front facing camera 652, rearfacing camera 654 and left and right side facing cameras 656 a and 656b, respectively. As shown, illuminator 658 is disposed above the frontcamera 652. Similarly, illuminator 660 is disposed above the rear camera654. Side cameras 656 a and 656 b are shown each having a respectiveilluminator 662 a, 662 b disposed above it. In an alternativeconfiguration, the illuminators could be disposed beneath the respectivecameras. In yet another example, the emitter(s) could be placed in otherlocations around the vehicle that are not co-located with the cameras,such as along the roof. For instance, an illuminator placed on the roofcould be used to illuminate the entire field of view of the camera for agiven side of the vehicle.

In these examples, a single illuminator module could be placed on anyside of the camera. However, there would be some amount of occlusion ofpart of the projected light. For instance, with a single illuminatormodule to the side of the camera, there can be a large occlusions oflight on the other side of the camera, which can be disadvantageous inlow light situations. With a single module above the camera, to reduceocclusion this could necessitate moving the module out above and forwardof the camera. In some locations around the vehicle, e.g., in the frontand the rear, but on the sides of the vehicle there may be constraintson vehicle width and potential impact of other side sensors. And with asingle module below the camera, upward illumination can be decreased dueto occlusion. This could impact the ability of the sensor suite toclassify nearby objects, such as a person standing next to the vehicle.Thus, it is desirable to put the illuminator module where the field ofview of the corresponding camera is the smallest, since this reduces thepotential for field of view occlusions. In situations where the verticalfield of view is smaller than the horizontal field of view, it may besuitable to place the illuminator on top of and/or beneath the camera.However, this may not be possible in some situations due to otherconstraints for the sensor suite, vehicle size, etc.

In view of this, FIG. 6C illustrates a third configuration 680, in whichpairs of illuminators are disposed to either side of each respectivecameras. As with FIGS. 6A-B, four cameras are shown around the vehicle:front facing camera 682, rear facing camera 684 and left and right sidefacing cameras 686 a and 686 b, respectively. As shown, a pair ofilluminators 688 a and 688 b are arranged on either side of front camera602. Similarly, a pair of saddle IR illuminators 690 a and 690 b arearranged on either side of rear camera 684. In this example, sidecameras 686 a and 686 b are shown each having a respective pair ofilluminator 692 a, 692 b or 694 a, 694 b disposed to the sides thereof.This arrangement helps to minimize occlusions that can occur when onlyone illuminator module is placed to the side, top, or bottom of thecorresponding camera. FIG. 7F, discussed below, shows one arraignmentfor the pair of side illuminators for each camera 682, 684 and 686.

In any of these configurations, the camera lens may be coatedhydrophobically to repel water. And the cameras may be placed along thevehicle's exterior so that the lenses are easily cleaned using anonboard cleaning system. Such features are discussed further below.

Perimeter Sensor Housings

According to aspects of the technology, housings with integrated sensorassemblies including multiple different sensors (e.g., lidar, camera,radar, etc.) can be located at various places along the vehicle. FIGS.1A-C illustrate exemplary housing placements for such integrated sensorassemblies. As discussed above, each location provides for specificcoverage around the vehicle from its sensors, which have particularfields of view. The arrangement of each sensor along the housing andrelative to the other sensors is important, as there may be significantbenefits (or drawbacks) to different arrangements. One or more of thesensor housing may have a close sensing camera assembly as describedabove. Several examples are discussed below.

In a first example shown in FIG. 7A, a sensor suite is arranged in aside perimeter housing along the left or right side of the vehicle infront of the driver or passenger side door. In particular, FIG. 7Aillustrates a view showing a first housing 700 a along the left frontquarterpanel and a second housing 700 b along the right frontquarterpanel. The housing 700 b may be a mirror image of the housing 700a. FIGS. 7B-F illustrate various views of a side perimeter housing 700.As shown in the perspective view of FIG. 7B and front view of FIG. 7C,the suite of sensors in the side perimeter housing 700 includes a lidarunit 702, a close sensing camera assembly 704, a radar unit 706, aforward-facing perimeter view camera 708 and a side-facing perimeterview camera 710.

As shown in FIGS. 7B-C, the radar unit 706 is disposed between the frontand side-facing cameras on the one side and the lidar and close sensingcamera assembly on the other side. Separation between the radar unit andthe aligned lidar and close sensing camera assembly avoids interferenceand potential occlusion.

The close sensing camera assembly 704 is disposed below the lidar unit702, for instance to enable object classification to supplement objectdetection by the lidar unit. While shown aligned below the lidar unit702, the camera of the close sensing camera assembly 704 may be locatedanywhere within approximately 0.25-0.4 m of the lidar unit 702. In orderto avoid parallax, which may adversely impact image classification, thecamera should be as close as possible to the lidar unit without creatingocclusions between the sensors. And while shown aligned below the lidarunit 702, the camera of the close sensing camera assembly 704 may bedisposed above the lidar unit 702. Either arrangement minimizes thelikelihood of occlusion and parallax. Spatial constraints of the housingunit and/or the vehicle's overall dimensions may also limit placement ofthe sensors relative to one another.

As shown in the left and right side views of FIGS. 7D and 7E,respectively, there is a separating surface 712 between the lidar unit702 and the close sensing camera assembly 704. The separating surfacemay be arranged at a downward sloping angle. The outward sloping surfaceallows water, snow, etc., to slide off, which minimizes the likelihoodof an obstruction or occlusion of the sensors. As the lidar sensor mayhave a limited view immediately beneath itself, aligning the cameraassembly directly below it the lidar helps with object detection ofpotentially lidar-occluded objects. For instance, as shown in the sideviews, the close sensing camera assembly 704 is angled downward to coverthe immediate vicinity around the vehicle.

The enlarged view of FIG. 7F illustrates that the assembly 704 includesa camera 714, a pair of illuminator modules 716 a and 716 b, and a setof cleaning mechanisms 718 a, 718 b and 718 c. Extensions 720 may beincluded that extend from the housing surface to ensure that there is noleakage of light into the lens of the camera 714. Each module 716 a and716 b may include one or more secondary lenses 722, which can beemployed to focus the light, e.g., IR light, along one or more desiredareas. By way of example, these secondary lenses 722 can increase thewidth of the field of view for the illuminator module. The cleaningmechanisms 718 may include fluid and/or forced air sprays to clean thecamera and/or illuminator modules. Alternatively or additionally, one ormore wipers (not shown) may be employed to keep the lenses clean.

FIG. 8 illustrates an example occlusion scenario 800. While the lidarsensor may have a wide coverage azimuth of, e.g., 180°, as shown it mayhave an occlusion region 802 immediately adjacent to the vehicle that isbeneath the lidar sensor, which illustrated as a shaded triangular area.Because the close sensing camera assembly is configured to supplementthe perception information obtained by the lidar sensor and is angleddownward, it is able to mitigate the lidar sensor's occlusion region802. For instance, if there is an object adjacent to the front tire, thecamera of the close sensing camera assembly is configured to detect it,as shown by the linear elements 804 within the shaded area. While it maynot be feasible for the camera to see within a few centimeters to theside of the vehicle, the camera is positioned so that an object thatclose to the vehicle is at least 50% visible. In one example, the cameramay have an azimuth field of view on the order of 170°-200°.

Returning to FIG. 7B, as noted above there is a forward-facing perimeterview camera 708 and a side-facing perimeter view camera 710 in theexemplary side perimeter housing 700. These cameras are configured toprovide front and side imagery with a minimum azimuth coverage as shownin example 900 of FIG. 9 . For instance, the side-facing camera 710 maybe configured to provide a minimum of +/−150 FOV 902 to the side of thevehicle, although it may provide up to +/−30-40° or more. In oneexample, there may be a minimum FOV 904 of 15° toward the rear of thevehicle, and a minimum FOV 906 of 25-35° toward the front of thevehicle. The front-facing camera(s) 708 may have an outer azimuth FOV908 on the order of 10-20°, and an inner azimuth FOV 910 on the order of20-40°. In some instances, the driver's side front-facing camera mayhave a wider FOV than the passenger's side front-facing camera, forinstance to provide increased visibility for left hand turns. Asillustrated in FIG. 7B, the front-facing camera 708 may be disposedhigher (or lower) than the side-facing camera 710. This may be done toaccommodate the various sensor units within the housing 700.

This pair of cameras may be used in conjunction with the other sensors,for instance to bolster radar detection and classification in difficultoccluded cross traffic and unprotected turn scenarios. In one scenario,the cameras 708 and 710 are not primarily used for close-in sensing, anduse ambient light without an IR illuminator. The side-facing perimeterview camera 710 may be located as far forward in the housing 700 orelsewhere on the vehicle to reduce the likelihood of being occludedwhile the vehicle inches into an intersection or is making a tight turn.The front-facing perimeter view camera 708 may also be located as farforward as possible to better see around occluding objects in front ofthe vehicle.

FIG. 10 illustrates an occlusion example for a turn scenario 1000. Inthis scenario, vehicle 1002 is preparing to make a left turn as shown bythe dashed arrow. Here, a truck 1004 or other object may occlude anothervehicle 1006 from the field of view of one of the sensors on the vehicle1002. For instance, a roof-mounted sensor 1008 may have a FOV 1010, thatis partially occluded in region 1012 by the truck 1004. However, aperimeter-facing camera 1014 has a different FOV 1016 that is able tosee at least part of the other vehicle 1006. The perimeter-facingcameras are beneficial in a wide variety of other scenarios, such asmaneuvering around another vehicle such as when there may be oncomingtraffic, seeing adjacent lanes rearward when there is an occludingvehicle behind the autonomous vehicle, when merging into high speedtraffic such as via an on-ramp of a freeway.

Another example of a perimeter housing assembly is shown in FIGS. 11A-C.In particular, FIGS. 11A-B illustrate a rear housing assembly 1100,which is shown in an example position 1110 on the rear fascia of a sedanor other vehicle in FIG. 11C. While the position 1110 is shown on therear left side of the vehicle, another rear housing assembly may also bedisposed on the right side of the vehicle. As indicated in FIGS. 11A-B,a first sensor 1102 and a second sensor 1104 are disposed in the housing1100. By way of example, the first sensor 1102 is a radar sensor and thesecond sensor is a camera. These sensors are able to provide informationabout other vehicles approaching from the rear, for example to accountfor high speed lane changes to the right or left.

As shown in example 1200 of FIG. 12 , the rear-facing perimeter viewcameras are configured to provide rear imagery with a minimum azimuthcoverage. For instance, the rear-facing camera 1100 may be configured toprovide between 30-60° FOV 1202 to the rear of the vehicle. By way ofexample, the rear-facing camera may have an outer azimuth on the orderof 15-35° (e.g., to see cars approaching in adjacent lanes), and aninner azimuth the order of 10-25° (e.g., to see following vehicles inthe same lane). FIG. 13 illustrates a scenario 1300, showing that therear-facing camera is able to see the car approaching in the adjacent(left) lane that would otherwise be occluded by the truck.

In another example shown in FIG. 14A, a front sensor housing 1400 isarranged along or adjacent to the front bumper, for instance to detectand classify objects directly in front of the vehicle. FIGS. 14B-Eillustrate various views of the front sensor housing 1400. As shown inthe perspective view of FIG. 14B and front view of FIG. 14C, the suiteof sensors in the front sensor housing 1400 includes a lidar unit 1402and a close sensing camera assembly 1404.

The close sensing camera assembly 1404 is disposed directly above thelidar unit 1402, for instance to enable object classification tosupplement object detection by the lidar unit. While shown aligned abovethe lidar unit 1402, the camera of the close sensing camera assembly1404 may be located anywhere within approximately 0.25-0.4 m of thelidar unit 1402. In order to avoid parallax, which may adversely impactimage classification, the camera should be as close as possible to thelidar unit without creating occlusions between the sensors. And whileshown above the lidar unit 1402, the camera of the close sensing cameraassembly 1404 may be disposed below the lidar unit 1402. Eitherarrangement minimizes occlusion. Spatial constraints of the housing unitand/or the vehicle's overall dimensions may also limit placement of thesensors.

As shown in the side view of FIG. 14D, there is a separating surface1406 between the lidar unit 1402 and the close sensing camera assembly1404. The separating surface may be arranged at an angle, e.g., to allowwater, snow, etc., to slide off, which minimizes the likelihood of anobstruction or occlusion of the sensors. Also shown in the side view,the close sensing camera assembly 1404 is angled downward to cover theimmediate vicinity around the vehicle. The enlarged view of FIG. 14Eillustrates that the assembly 1404 includes a camera 1408, a pair ofilluminator modules 1410 a and 1410 b, and a set of cleaning mechanisms1412 a, 1412 b and 1412 c. Extensions 1414 may be included that extendfrom the housing surface to ensure that there is no leakage of lightinto the lens of the camera 1408. As shown, each module 1410 a and 1410b may include one or more secondary lenses 1416, which can be employedto focus the light along one or more desired areas. The cleaningmechanisms 1412 may include fluid and/or forced air sprays to clean thecamera and/or illuminator modules.

FIG. 15 illustrates a variation 1500 of front sensor housing 1400. Inthis variation, the housing 1500 includes lidar unit 1402 and a closesensing camera 1502, which omits the IR illumination modules andcleaning mechanisms. In another variation without illumination, thecleaning mechanisms can be included. Similarly, in a further variationwithout cleaning mechanisms, illumination can be included.

In another example shown in FIG. 16A, a rear sensor housing 1600 isarranged along or adjacent to the rear bumper, for instance to detectand classify objects directly behind the vehicle. FIGS. 16B-E illustratevarious views of the rear sensor housing 1600. As shown in theperspective view of FIG. 16B and front view of FIG. 16C, the suite ofsensors in the front sensor housing 1600 includes a lidar unit 1602 anda close sensing camera assembly 1604.

The close sensing camera assembly 1604 is disposed directly above thelidar unit 1602, for instance to enable object classification tosupplement object detection by the lidar unit. While shown aligned abovethe lidar unit 1602, the camera of the close sensing camera assembly1604 may be located anywhere within approximately 0.25-0.4 m of thelidar unit 1602. In order to avoid parallax, which may adversely impactimage classification, the camera should be as close as possible to thelidar unit without creating occlusions between the sensors. And whileshown above the lidar unit 1602, the camera of the close sensing cameraassembly 1604 may be disposed below the lidar unit 1602. Eitherarrangement minimizes occlusion. Spatial constraints of the housing unitand/or the vehicle's overall dimensions may also limit placement of thesensors.

As shown in the side view of FIG. 16D, there is a separating surface1606 between the lidar unit 1602 and the close sensing camera assembly1604. The separating surface may be arranged at an angle, e.g., to allowwater, snow, etc., to slide off, which minimizes the likelihood of anobstruction or occlusion of the sensors. Also shown in the side view,the close sensing camera assembly 1604 is angled downward to cover theimmediate vicinity around the vehicle. The enlarged view of FIG. 16Eillustrates that the assembly 1604 includes a camera 1608, a pair ofilluminator modules 1610 a and 1610 b, and a set of cleaning mechanisms1612 a, 1612 b and 1612 c. Extensions 1614 may be included that extendfrom the housing surface to ensure that there is no leakage of lightinto the lens of the camera 1608. As shown, each module 1610 a and 1610b may include one or more secondary lenses 1616, which can be employedto focus the light along one or more desired areas. The cleaningmechanisms 1612 may include fluid and/or forced air sprays to clean thecamera and/or illuminator modules.

FIG. 17 illustrates a variation 1700 of front sensor housing 1600. Inthis variation, the housing 1700 includes lidar unit 1602 and a closesensing camera 1702, which omits the illumination modules and cleaningmechanisms. In another variation without illumination, the cleaningmechanisms can be included. Similarly, in a further variation withoutcleaning mechanisms, illumination can be included.

As noted above, the close sensing cameras of the various sensor housingsare arranged at a downward angle. By way of example, they may have adownward angle on the order of 20-40° to maximize lower field of viewcoverage and cover as much of the counterpart lidar's FOV as possible,because these cameras bolster the lidars' detection and classification.While different arrangements have been shown for co-location of theclose sensing camera assemblies and lidar units, in general each camerais placed relative to its lidar unit so that it minimizes occlusions tothe lidar in all instances.

Sensor cleaning is important to proper, effective operation of theperception system. There are different options for cleaning varioussensors as the vehicle is operating in an autonomous driving mode. Forinstance, the cleaning system may spray a cleaning fluid onto a camera(and IR emitter), use a wiper, and/or an air puffer. A sprayer or othercleaning unit may be at a fixed position relative to the sensor, or maybe configured to telescope out in order to clean the unit on anas-needed basis. The cleaning unit should be arranged to avoid occlusionof any of the sensors in the sensor housing or otherwise impact sensorFOV. For instance, the sprayer tips of cleaning mechanisms 718, 1412 and1612 are positioned as such to not occlude the cameras and not reflectlight back into the cameras.

In addition to cleaning, sensors may be conditioned, for instance byproviding heat to eliminate condensation or frost from a camera or othersensor. By way of example, defrosting heaters may be positioned alongthe front window element of each perimeter view camera, such as aheating element sandwiched between the front glass and the housing ofthe camera unit.

A shroud or other structure can be employed to limit dirt and otherforeign objects from covering the sensor. However, as with the cleaningelements, the shroud should not occlude the sensors or otherwise impacttheir FOV. In the case of a camera or illumination emitter, ahydrophobic coating may be applied to the glass or plastic cover tominimize moisture accumulation.

The positions of the sensors within the housing and along the vehicleshould be considered when selecting the type and placement of cleaningunit. By way of example, it may be hard to clean a camera located in theside mirror assembly. For instance, it may be challenging to spray fluidonto a particular spot, or there may be limits on how to route fluidthrough the vehicle to cleaning unit (e.g., if the cleaning unit islocated on-door v, off-door). The type of cleaning mechanism may bechosen in accordance with how important that sensor is for autonomousdriving. By way of example, cleaning of the front sensor housing unitmay be more critical than cleaning of the rear sensor housing unit,because it may be determined that a clear view of leading vehicles ismore relevant to certain driving operations than a view of trailingvehicles. As such, redundant (e.g., spray system plus a wiper) cleaningmodules may be employed for more critical sensor housings. For othersensor housing, there may be no redundant cleaning mechanism. In thiscase, the cleaning mechanism may only be actuated at certain speeds(e.g., below 35-45 mph) or when the vehicle is stationary, becausecleaning may be less time sensitive than for other sensor housings.

FIG. 18 illustrates a flow diagram 1800 of a method in accordance withcertain actions described above. At block 1802, a control system of avehicle configured to operate in an autonomous driving mode imitatesoperation of a lidar sensor of a perception system of the vehicle, inorder to obtain lidar data within a threshold distance in a regionaround the vehicle. For instance, the threshold distance may be within1-3 meters from the vehicle. At block 1804, the control system initiatesimage capture by an image sensor of the perception system prior to thevehicle performing a driving action. The image sensor is disposedadjacent to the lidar sensor and arranged along the vehicle to have anoverlapping field of view of the region around the vehicle within thethreshold distance. The image sensor provides a selected resolution forobjects within the threshold distance. At block 1806, the control systemreceives the lidar data from the lidar sensor and the captured imageryfrom the image sensor. This may be done concurrently or sequentially. Atblock 1808, the control system processes the lidar data to detect anobject in the region within the threshold distance of the vehicle. Atblock 1810, the control system processes the captured imagery toclassify the detected object. And at block 1812, the control systemdetermines whether to cause one or more systems of the vehicle toperform the driving action based on classification of the detectedobject.

Unless otherwise stated, the foregoing examples and embodiments are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples or embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements. Theprocesses or other operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

The invention claimed is:
 1. An external sensing module for a vehicleconfigured to operate in an autonomous driving mode, the externalsensing module comprising: a cover configured to be affixed along afront section of the vehicle; a lidar sensor engaged with the coveralong a first section thereof, the lidar sensor being configured todetect objects in a region of an external environment around the vehicleand within a threshold distance of the vehicle; and an image sensorengaged with the cover along a second section thereof, the image sensorbeing configured for arrangement along the front section of the vehiclebetween a front bumper and a front windshield of the vehicle, the imagesensor having a selected resolution that enables classification of anobject detected by the external sensing module within the thresholddistance of the vehicle as a selected shape based on a minimum number ofpixels.
 2. The external sensing module of claim 1, wherein the imagesensor is arranged above the lidar sensor.
 3. The external sensingmodule of claim 1, wherein the image sensor is arranged below the lidarsensor.
 4. The external sensing module of claim 1, wherein the imagesensor is arranged at a downward angle relative to the front section ofthe vehicle, the downward angle selected to provide coverage within thethreshold distance of the vehicle.
 5. The external sensing module ofclaim 1, further comprising a cleaning device at least partly disposedwithin the cover, the cleaning device configured to provide cleaning toat least one of the image sensor or the lidar sensor.
 6. The externalsensing module of claim 5, wherein the cleaning device is disposedadjacent to the image sensor along the external sensing module.
 7. Theexternal sensing module of claim 1, further comprising one or moreilluminator units configured to illuminate a field of view of the imagesensor.
 8. The external sensing module of claim 7, wherein the one ormore illuminator units are disposed adjacent to the image sensor.
 9. Theexternal sensing module of claim 1, wherein the cover has a frontsurface that is at least partly curved to project outwardly from thefront section of the vehicle.
 10. The external sensing module of claim1, further comprising a radar sensor disposed along the external sensingmodule.
 11. The external sensing module of claim 1, wherein the coverincludes an opening adapted to receive a lens of the image sensor. 12.An external sensing module for a vehicle configured to operate in anautonomous driving mode, the external sensing module comprising: a coverconfigured to be affixed along a rear section of the vehicle; a lidarsensor engaged with the cover along a first section thereof, the lidarsensor being configured to detect objects in a region of an externalenvironment around the vehicle and within a threshold distance of thevehicle; and an image sensor engaged with the cover along a secondsection thereof, the image sensor being configured for arrangement alongthe rear section of the vehicle between a rear bumper and a rear windowof the vehicle, the image sensor having a selected resolution thatenables classification of an object detected by the external sensingmodule within the threshold distance of the vehicle as a selected shapebased on a minimum number of pixels.
 13. The external sensing module ofclaim 12, wherein the image sensor is arranged at a downward anglerelative to the rear section of the vehicle, the downward angle selectedto provide coverage within the threshold distance of the vehicle. 14.The external sensing module of claim 12, further comprising a cleaningdevice at least partly disposed within the cover, the cleaning deviceconfigured to provide cleaning to at least one of the image sensor orthe lidar sensor.
 15. The external sensing module of claim 14, whereinthe cleaning device is disposed adjacent to the image sensor along theexternal sensing module.
 16. The external sensing module of claim 12,further comprising one or more illuminator units configured toilluminate a field of view of the image sensor.
 17. The external sensingmodule of claim 12, wherein the cover has a front surface that is atleast partly curved to project outwardly from the rear section of thevehicle.
 18. The external sensing module of claim 12, further comprisinga radar sensor disposed along the external sensing module.
 19. Theexternal sensing module of claim 12, wherein the cover includes anopening adapted to receive a lens of the image sensor.
 20. A vehicleconfigured to operate in an autonomous driving mode, comprising: a frontsection having a front bumper and a front windshield; a rear sectionhaving a rear bumper and a rear window; a driving system including: adeceleration system configured to control braking of the vehicle; anacceleration system configured to control acceleration of the vehicle;and a steering system configured to control wheel orientation and adirection of the vehicle; a first external sensing module disposedbetween the front bumper and the front windshield, the first externalsensing module comprising: a first cover configured to be affixed alongthe front section of the vehicle; a first lidar sensor engaged with thecover along a first section thereof, the first lidar sensor beingconfigured to detect objects in a first region of an externalenvironment around the vehicle and within a threshold distance of thevehicle; and a first image sensor engaged with the first cover along asecond section thereof, the first image sensor having a selectedresolution that enables classification of a first object detected by thefirst external sensing module within the threshold distance of thevehicle as a selected shape based on a minimum number of pixels; and asecond external sensing module disposed between the rear bumper and therear window, the second external sensing module comprising: a secondcover configured to be affixed along the rear section of the vehicle; asecond lidar sensor engaged with the second cover along a first sectionthereof, the second lidar sensor being configured to detect objects in asecond region of the external environment around the vehicle and withinthe threshold distance of the vehicle; and a second image sensor engagedwith the second cover along a second section thereof, the second imagesensor having the selected resolution that enables classification of asecond object detected by the second external sensing module within thethreshold distance of the vehicle as the selected shape based on theminimum number of pixels.